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Executive  Summary 


This  report  outlines  the  approach,  schedule  and  responsibilities  for  a  preliminary  user  trial  of  the 
Situation  Awareness  Module  (SAM)  of  the  Situation  Awareness  System  (SAS)  during  a  battle  group 
training  exercise  to  be  conducted  in  Gagetown  1-9*  May  1999.  Training  is  the  main  priority  for  the 
exercise  and  all  data  capture  for  this  trial  must  fit  in  with  that  priority.  Fourteen  SAM  systems  and 
26  Precision  Lightweight  Global  Positioning  System  Receivers  (PLGR)  will  be  distributed  within  the 
unit  command  structure.  The  main  focus  of  the  trial  will  be  on  ease  of  use  and  usefulness  of  SAM. 
Observers  will  be  teamed  in  pairs,  one  pair  attached  to  battle  group  HQ  and  to  each  of  two  combat 
teams.  Data  capture  will  employ  direct  observation,  ad  hoc  interviews,  formal  questionnaires  and  a 
Focus  Group  after  the  exercise.  Annexes  to  the  report  provide  details  of  questionnaires  and  other 
data  capture  approaches.  For  data  capture  to  be  successful,  all  observers  will  require  familiarization 
with  SAM,  the  exercise,  and  data  capture  procedures. 
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1.  Introduction 


1.1.  Outline 

This  document  provides  the  basic  information  and  materials  to  prepare  for,  conduct  and  analyze  data 
from  a  Preliminary  User  Trial  of  the  Situation  Awareness  Module  (SAM)  of  the  Situation 
Awareness  System  (SAS).  The  Plan  is  to  be  used  during  a  Battle  Group  level  field  trial  of  SAM  - 
Royal  Canadian  Dragoon  (RCD)  Battle  Group  (BG)  Exercise  in  May  1999. 

1.2.  Background 

The  Position  Determination  &  Navigation  Project  (PDALF)  entered  implementation  in  1996  with  the 
mandate  to  deliver  a  Global  Positioning  System  (GPS)  capability  for  the  army  followed  by  a  vehicle 
mounted  and  integrated  “Blue  Force”  Situation  Awareness  (SA)  capability  riding  on  the 
communications  backbone  provided  by  the  Tactical  Command,  Control  and  Communication  System 
(TCCCS)  Project.  The  GPS  capability  has  been  fielded  for  dismounted  use  and  the  SA  capability  is 
currently  in  implementation  with  fielding  scheduled  to  commence  in  Oct  2000.  Concurrent  with  this 
project  there  has  been  a  shift  in  the  Canadian  Army’s  doctrinal  focus  such  that  significant  effort  is 
now  being  placed  on  the  development  of  concepts,  procedures  and  tools  to  assist  commanders  in 
making  better  decisions,  faster.  The  segment  of  the  PDALF  project  that  will  deliver  the  SA 
component  is  known  as  the  SAS.  SAS  includes  both  modifications  in  TCCCS/IRS  and  SAM.  To 
date,  this  system  has  been  developed  with  user  input  in  a  lab  type  environment.  It  is  now  required 
that  the  SAM  be  evaluated  in  field  environment  prior  to  the  close  of  the  development  cycle. 

Accordingly,  a  limited  preliminary  user  field  trial  will  be  conducted  in  conjunction  with  a  RCD  BG 
training  exercise  to  be  conducted  in  Gagetown  in  early  May  but  the  preliminary  user  trial  of  the 
SAM  must  defer  to  the  training  objectives  of  the  exercise.  This  report  outlines  a  trial  plan  to  assess 
the  usability  and  utility  of  the  SAM  using  data  collected  through  observation,  interviews, 
questionnaires,  and  Focus  Groups.  As  a  lower  priority,  opportunities  will  be  sought  to  pilot 
methods  for  capturing  performance  data  (speed  and  accuracy)  using  the  SAM  data  recording  system. 

The  plan  is  being  prepared  under  contract  to  the  Defense  and  Civil  Institute  of  Environmental 
Medicine  (DCIEM)  on  behalf  of  the  Director  of  Land  Requirements  (DLR  4-9) 

1.3.  Aim 

The  aim  of  this  project  is  to  develop  a  trial  plan  to  evaluate  the  usability  and  utility  of  SAM  at  the 
battle  group  level  under  exercise  conditions. 

1 .4.  Report  Format 

The  main  body  of  the  report  provides  sections  on  background,  scope,  participants,  and  approach. 
Annexes  provide  details  of  the  exercise  scenario,  data  capture  instruments  (questionnaires,  etc),  and 
SAM  software  functionality. 
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1.5.  Scope 

The  trial  plan  will  be  limited  to  the  assessment  of  usability  and  utility  of  SAM  with  in  the  context  of  a 
Battle  Group  training  exercise.  However,  where  possible,  opportunities  will  also  be  sought  to  gather 
data  in  order  to  evaluate  the  SAM  training  package  provided  as  well  as  data  concerning  tactics, 
techniques  and  procedures  (for  the  use  of  SAM). 

Utility  -  is  defined  as  the  degree  to  which,  independent  of  ‘  Usability ’,  SAM  functions  are  useful  for 
user  goal  and  task  related  SA  needs 

Usability  -  is  defined  as  the  degree  to  which,  independently  of  ‘Utility' ,  SAM  features  are  easy  to 
use  in  terms  of: 

•  cognitive  fit  (information  content  and  display  format  in  relation  to  cognitive  demands). 

•  physical  fit  (users  sensory,  anthropometric,  and  bio-mechanical  needs). 

For  Usability,  priority  will  be  given  to  cognitive  fit.  Both  utility  and  usability  will  be  examined  in 
terms  of  exercise  mission  related  goals  and  tasks  as  well  as  SAM  functionality. 

Assessment  of  SA  will  be  limited  to  participants’  perception  of  some  key  aspects  of  their  SA  including 
location  of  friendly  forces,  relevant  unit  status  i.e.  the  information  provided  by  the  SAM  software. 

SA  will  be  assessed  in  terms  of  users’  ability,  in  a  timely,  complete  and  accurate  manner,  to: 

•  Detect  pertinent  information  elements. 

•  Integrate  the  information  detected  into  a  picture  of  the  current  situation. 

•  Project  the  current  situation  with  respect  to  friendly  forces  into  the  near  future. 

Where  possible,  the  impact  of  using  SAM  will  be  assessed  in  terms  of: 

•  Perceived  quality  of  decision  making. 

•  Patterns  of  communication. 

•  Overall  workload. 

Several  types  of  SAM  units  will  be  deployed  including  14  (and  one  spare)  fully  functional  SAMs  and 
26  PLGR  systems.  However,  this  trial  will  be  limited  to  assessment  of  the  14  SAM  systems. 
Deployment  of  these  systems  is  detailed  later  in  the  report. 

1.6.  Limitations 

•  Data  capture  must  not  interfere  with  unit  training  goals.  Data  capture  methods  must  be 
easy  to  apply  and  readily  adaptable  to  the  unpredictable  and  changing  circumstances  of  a 
training  exercise  in  terms  of  schedule  and  access  to  personnel.  Access  to  SAM  users  for 
data  capture  will  need  to  be  coordinated  with  the  relevant  unit  commander  on  the 
ground. 

•  The  availability  of  observers  during  the  field  trial  is  uncertain.  Observers  will  require 
training  and  briefing  prior  to  data  capture. 

•  SAM  equipment  provided  is  prototype  and  still  under  development  and  some  functions 
may  not  be  entirely  reliable.  Ruggedization  is  limited. 

•  The  small  number  of  trial  participants,  especially  within  specific  BG  and  Combat  Team 
(cbt  tm)  member  positions,  will  limit  analysis  to  descriptive  statistics. 

•  Unit  organization  and  SAM  distribution  are  subject  to  change  up  to  and  during  the  trial. 
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•  SAM  installation  in  vehicles  may  compromise  best  use,  especially  when  mobile.  If  so, 
this  will  need  to  be  taken  into  account  when  interpreting  results. 

•  The  trial  sample  comprises  a  cross  section  of  combat  team  members  with  potentially 
different  needs.  The  collection  and  aggregation  of  data  should  take  this  into  account. 

•  Exercise  scenarios  will  not  be  replicated  and  no  experimental  controls  will  be  available. 
This  means  that  SAM-present  vs  SAM  absent  conditions  cannot  be  compared  under 
repeatable  conditions. 

1.7.  SAM  Software  Functionality  (Annex  A) 

Broadly  speaking,  SAM  enables  users  to: 

•  Log  on  and  configure  the  overall  system. 

•  Set  up  and  maintain  an  electronic  map  showing  location  of  friendly  (automatically  using 
GPS)  and  enemy  units  (manually  by  contact  reports),  including  functions  related  to 
ORBAT,  unit  aggregation  and  status,  track  history,  etc. 

•  Create,  send,  receive,  and  manage  text  messages,  reports,  returns,  alerts,  etc. 

•  Create,  send,  receive,  and  manage  plans  and  orders  in  various  formats. 

1.8.  Participating  Units  (Annex  B) 

Units  participating  in  the  trial  will  include  an  ad  hoc  BG  comprising  RCD  Regimental  (Regt)  Head 
Quarters  (HQ),  HQ  Squadron  (Sqn),  B  Sqn,  B  Company  (Coy)  &  C  Coy  1st  Battalion,  Royal 
Canadian  Regiment  (1  RCR)  with  other  elements  as  shown  in  Annex  B.  In  addition  there  will  be  an 
artillery  battery  commander  and  Forward  Observation  Officer  (FOO)  personnel  and  one  engineer 
support  troop,  organized  as  combat  teams.  All  relevant  exercise  personnel  will  be  fully  trained  in 
SAM  functionality  and  procedures  prior  to  the  exercise.  Some  elements  of  the  exercise  will  be 
notional  and  will  therefore  not  be  equipped  with  SAM. 

1.9.  Distribution  of  SAM  Systems  (Annex  B) 

Several  types  of  SAM  units  will  be  distributed  but  only  the  14  (and  one  spare)  fully  functional  SAM 
systems  will  be  assessed  during  this  trial.  The  distribution  of  the  SAM  systems  to  different  call 
signs  (c/s)  is  shown  in  Annex  B. 

1.10.  Exercise  Preparation  and  Schedule  (Annex  C) 

RCD  BG  Exercise  is  a  battle  group  exercise  intended  as  a  work  up  for  the  Combat  Team 
Commanders  course  exercise  comprising  an  advance  to  contact  preceding  a  hasty  attack  followed  by 
a  defensive  action.  The  exercise  runs  from  1-9*  May  1999  and  is  expected  to  follow  the  schedule 
outlined  below.  In  addition,  there  will  be  an  RCD  SAM  familiarization  exercise  on  12-14  April  in 
Petawawa.  Observers  will  be  briefed  by  the  Trial  Director  on  27  April  in  Gagetown  during 
coordination  briefings  (trial  outline,  exercise  scenario,  data  capture,  responsibilities,  timings) 
Infantry  units  and  other  users  not  yet  familiar  with  SAM  will  be  trained  on  29-30  April  in 
Gagetown.  It  may  be  possible  to  familiarize  observers  with  SAM  at  the  same  time.  (Observers 
require  sufficient  knowledge  to  recognize  SAM  functions,  not  the  skill  to  operate  or  advise  users.) 
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The  approximate  sequence  of  events  for  the  data  collection  team  is  outlined  below.  Implementation 
is  the  responsibility  of  the  Trial  Director. 

•  Meet  and  brief  trial  participants  prior  to  the  exercise  (as  determined  by  Trial  Director). 

•  Phase  I  exercise  data  capture  followed  by  data  collation  and  team  meeting  (May  1-2). 

•  Phase  II  exercise  data  capture  followed  by  data  collation  and  team  meeting  (May  3-7). 

•  Phase  III  Focus  Group  data  capture,  final  data  collation  (May  8-9). 

•  Data  reduction  and  analysis  (as  determined  by  Trial  Director). 

•  Report  writing  and  submission  (as  determined  by  Trial  Director). 


DATE 

Activity 

Saturday 

1st  May 

Exercise  commences 

mm 

Independent  coy  and  sqn  training. 

Observers  coordinate  with  SAM  users.  Limited  data  capture. 

End  of  day  orders:  cbt  tm  advance  to  contact  within  BG  context  on  3-4*  May. 
Coordinate  with  SAM  users  for  observation  of  different  c/s  and  ad  hoc  interviews. 

■ 

Advance  to  contact 

Repeat  3-5  quick  attacks  per  day  using  North-South  axis  of  Lawfield  Corridor. 

May  include  an  attack  on  bride  at  the  South  end  of  the  Lawfield  Corridor  towards  the 
end  of  day  4.  Battle  Procedure  hides  at  approximately  2300  hrs. 

Coordinate  with  SAM  users  to  complete  interim  questionnaire  on  completion. 

End  of  day  orders:  hasty  defence  and  delay  using  the  Jerusalem  Ridge  on  5-6*  May. 
Coordinate  with  SAM  users  for  observation  of  different  c/s  and  ad  hoc  interviews. 

545th  May 

Conduct  hasty  defence  and  delay  (North  end  of  Lawfield  Corridor). 

Likely  followed  by  pursuit  on  day  6. 

Coordinate  with  SAM  users  to  complete  interim  questionnaire  on  completion. 

7th  May 

Demonstration  Rehearsal  to  include  a  BG  attack  on  Headline  Ridge. 

This  will  be  at  least  two  hasty  attacks, 

one  with  obstacles  and  the  other  without  obstacles. 

Limited  data  capture.  Coordination  mtg  for  observer  teams.  Plan  Focus  Group 

Maintenance  and  contingency  for  exercise  slippage. 

Main  Focus  Group(s)  for  user  trial  (2-3  hours). 

m 

Visitors  day,  and  maintenance. 

Spare.  Ad  hoc  interviews  in  selected  areas,  if  required. 

Table  1:  Exercise  Schedule 
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2.  Method 


2.1.  Outline  Approach 

Based  on  discussions  with  the  Trial  Director  and  Unit  OPI,  data  capture  during  the  BG  exercise  has 
been  divided  into  three  phases. 

•  Phase  I:  Sub-unit  work  up  exercises  with  breaks. 

-  Observation,  ad  hoc  interviews,  ad  hoc  performance  data,  mini  Focus  Groups. 

•  Phase  II:  Battle  group  24/7  exercise  with  limited  breaks,  consistent  with  the  training 
objectives. 

-  Observation,  ad  hoc  performance  data  mini  Focus  Groups. 

•  Phase  III:  Post  exercise: 

-  Exit  Questionnaire 

-  Exit  Focus  Groups. 

Throughout  each  phase,  data  capture  will  concentrate  on  three  unit  groups:  BG  or  Regt  HQ,  and  two 
combat  teams  (see  Figure  1).  Observers  will  work  in  teams  of  two:  one  army  officer  and  one 
defence  scientist.  Each  Observation  Team  will  be  responsible  for  data  capture  within  one  unit 
group.  Overall  coordination  will  be  the  responsibility  of  the  trial  director.  Opportunities  for 
Observers  to  accompany  c/s  during  the  exercise  is  to  be  coordinated  with  RCD  BG  HQ  and  Coy/Sqn 
commanders. 


Figure  1:  Observer  Organization 
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Observers  will  be  required  to  complete  the  tasks  listed  below.  Observers  should  be  thoroughly 
briefed  on  the  use  of  the  data  sheets  provided  with  this  trial  and  familiarized  with  the  general 
principles  of  capturing  observational  data,  as  well  as  conducting  interviews  and  Focus  Groups. 
Depending  on  their  familiarity  with  army  exercises,  observers  should  also  be  briefed  on  unit 
organization,  exercise  scenarios,  and  the  general  conduct  of  army  operations  in  relation  to  likely 
SAM  usage.  For  the  unit  of  interest,  the  role  of  each  Observation  Team  will  be  to: 

•  record  conditions  of  SAM  use. 

•  watch  and  record  how  participants  use  SAM  to  accomplish  their  tasks. 

•  elicit  and  record  ad  hoc  comments  from  SAM  users  while  using  SAM. 

•  interview  SAM  users  at  appropriate  points  during  the  exercise. 

•  administer  questionnaires  at  appropriate  times. 

•  prepare  for  and  conduct  an  exit  Focus  Group. 

•  ensure  all  SAM  users  complete  the  required  questionnaires. 

•  safeguard  and  submit  completed  data  sheets  to  the  Trial  Director  for  later  analysis. 

Observers  should  also  seek  opportunities  to  time  performance  of  selected  tasks. 

Although  observers  will  not  be  able  to  cover  each  SAM  module  at  all  times,  they  will  be  expected  to 
observe  and  record  a  cross  section  of  users  and  SAM  functions  using  the  SAM  function  checklist 
provided  (Annex  A). 

Annexes  provide  the  following: 

A.  SAM  outline  functionality 

B.  Trial  participants 

C.  Exercise  schedule 

D.  Observers’  data  package. 

E.  Mid  trial  questionnaires 

F.  End  trial  questionnaire  and  Focus  Group. 

G.  Archival  data  capture 

2.2.  Trial  Participants  (Annex  B) 

For  the  purpose  of  this  trial  plan,  Trial  Participants  are  the  users  of  the  14  (and  one  spare)  full 
function  SAM  systems.  Trial  participants  represent  a  cross  section  of  BG  and  cbt  tm  command 
personnel  at  the  BG  HQ,  Sqn/Coy  level  from  the  infantry,  armour,  and  artillery.  Annex  B  shows 
details  and  organization  of  Trial  Participants. 

All  participants  will  have  received  training  in  SAM  from  the  software  developer  prior  to  the  start  of 
the  exercise  though  attendance  at  the  training  and  its  effectiveness  remain  to  be  established.  RCD 
participants  will  also  have  benefited  from  familiarization  with  SAM  in  the  field.  Infantry  and  other 
units  will  only  have  had  limited  opportunity  to  prepare  for  the  use  of  SAM  for  this  exercise. 
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2.3.  Data  Collection  (Annex  D,  E,  F,  G) 

Data  collection  approaches  are  based  on  exercise  phases  and  scenario  events.  Data  collection 
approaches  are  outlined  in  Table  2  below.  A  coordination  meeting,  as  directed  by  the  Trial 
Director,  of  all  observers  should  be  held  between  each  phase  to  discuss  modifications,  cross  check 
data  and  to  store  data  acquired  in  a  safe  place.  Waterproof  data  collection  envelopes  should  be 
provided  for  each  exercise  phase  for  each  Observation  team. 


Observer  teams 

HQ  Cbt  tm  A  Cbt  tm  B 

Phase  1 
Independent 
Coy/Sqn 
work  up 

•  Direct  observation 

•  Ad  hoc  Interviews 

•  Ad  hoc  data  capture 

•  Mini-Focus  Groups 

Phase  II 

Cbt  Tm 
within  BG 
context 
scenario 

•  Mini-Focus  Groups 

•  Short  questionnaire 

•  Direct  observation 

•  Ad  hoc  Interviews 

•  Ad  hoc  performance  data 

•  Archival  message  data  (if  available) 

Phase  III 

After  exercise 
activity 

•  Major  Focus  Groups 

•  Exit  questionnaire 

•  Archival  message  analysis 

Table  2:  Data  Capture  Approaches 


Observation  teams  should  coordinate  with  the  appropriate  unit  commander  so  that  all  SAM  users  are 
interviewed  as  well  as  all  key  SAM  functions  observed.  A  checklist  of  SAM  functionality  can  be 
seen  in  Annex  A. 

2.3.1.  Direct  observation  (see  Annex  D) 

During  observation,  observers  will  use  record  incidence  of  SAM  function  use  by  exercise 
event,  and  record  data  as  available  according  to  the  Observers  data  capture  sheet.  This 
includes  exercise  phase,  task  being  performed,  conditions  (night  /  day,  mobile/static,  etc), 
observers  and  users  comments,  ratings  and,  where  appropriate  and  possible,  time  to  perform 
selected  tasks.  Each  observation  team  may  carry  a  camera  to  provide  contextual  pictures  to 
supplement  these  notes. 

2.3.2.  Ad  hoc  interviews  (see  Annex  D) 

During  the  combat  team  and  battle  group  phases  of  the  exercise,  observers  will  take  advantage 
of  lulls  in  the  exercise  tempo  to  question  users  about  SAM  functions  they  have  used  in  terms 
of  ease  of  use  and  usefulness. 
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2.3.3.  Ad  hoc  performance  data  (see  Annex  D) 

During  the  combat  team  and  battle  group  phases  of  the  exercise,  observers  will  take  advantage 
of  opportunities  to  identify  start  and  stop  points  for  selected  SAM  functions  and  to  time-task 
performance. 

2.3.4.  Mini  Focus  Groups  (see  Annex  F) 

During  the  mid  exercise  break  after  combat  team  exercises  and  before  the  full  battle  group 
scenario,  each  observer  team  should  bring  together  or  visit  each  sub  group  of  SAM  users  and 
conduct  a  short  Focus  Group  to  review  the  SAM  functions  used  up  to  that  point.  These  will 
capture  users’  comments  and  perception  of  ease  of  use  and  usefulness  by  exercise  task. 

2.3.5.  Mid  trial  &  exit  questionnaire  (see  Annex  E) 

During  the  first  phase  of  the  exercise  itself,  or  if  this  proves  impossible,  during  the  mini-Focus 
Groups  conducted  during  the  mid-exercise  break,  trial  participants  should  complete  the  short 
questionnaire  provided  for  the  different  stages  of  the  exercise.  Before  the  main  Exit  Focus 
Group  starts  at  the  end  of  the  exercise,  each  participant  should  complete  their  final  Exit 
Questionnaire  covering  the  exercise  as  a  whole. 

Collusion  should  be  prevented.  Observer  teams  should  ensure  that  trial  participants  should 
complete  and  submit  their  exit  questionnaire  separately  if  they  cannot  attend  the  main  post 
exercise  Focus  Group. 

The  Questionnaire  will  be  used  to  assess  overall  ease  of  use  and  usefulness  of  SAM,  to 
compare  it  with  the  existing  system,  to  assess  the  training  package  provided  prior  to  the  trial, 
and  to  identify  implications  for  tactics  and  standard  operating  procedures. 

2.3.6.  Exit  Focus  Group  (see  Annex  F) 

After  the  exercise,  all  trial  participants  should  be  brought  together  at  Petersville  Camp  in 
Gagetown  Training  Area  {time  to  be  confirmed  by  trial  director  and  RCD  Ops  O )  in  their 
participant  groupings  (i.e.  HQ,  Cbt  tm  A  and  B)  to  discuss  their  experience  with  SAM.  This 
Focus  Group  will  require  at  least  two  hours  and  should  be  conducted  under  shelter  and  away 
from  distractions.  A  class  room  environment  would  be  ideal,  though  not  essential.  One 
observer  should  facilitate  discussion  while  the  other  acts  as  a  recorder.  Focus  Group  format 
should  combine  SAM  functionality  with  exercise  sequence  to  review  users  experiences 
systematically  using  probe  questions  (see  Annex).  Some  probe  questions  are  also  suggested  in 
the  Annex.  Others  may  arise  as  a  result  of  observer  experience  during  the  exercise.  The  trial 
director  should  coordinate  consistency  of  questions  across  Focus  Groups.  Basic  rules  for 
Focus  Group  conduct  are  outlined  in  the  Annex.  These  are  designed  to  maximize  participant 
responses  and  minimize  any  chilling  effect  resulting  from  judgmental  or  dismissive  statements 
from  authority  figures.  Members  of  the  SAM  design  team  should  not  be  permitted  to 
participate  during  the  Focus  Group  though  observation  and  discussion  with  participants 
afterwards  are  encouraged. 

At  the  discretion  of  the  trial  director,  a  short,  second  focus  re-group  might  be  held  which  re¬ 
organizes  trial  participants  according  to  military  function  (Armour,  Infantry,  Artillery,  etc)  to 
provide  further  insight.  Time  permitting,  SAM  users  may  also  be  asked  to  speculate  on  the 
use  of  SAM  for  other  operations  than  those  exercised. 
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2.3.7.  Archival  message  data  (see  Annex  G) 

SAM  has  some  functions  that  save  incoming  and  outgoing  message  traffic  together  with 
distribution  lists  and  time  of  arrival.  Data  capture  should  be  attempted  using  these  functions 
for  each  full  service  SAM  c/s  and  archived  for  later  analysis.  These  functions  should  be  tried 
during  one  scenario  replication  during  Phase  I  of  the  exercise  to  ensure  that  the  approach 
works  effectively.  If  successful  so,  data  capture  should  be  considered  for  selected  exercise 
scenarios  during  the  remainder  of  the  exercise.  It  will  be  important  for  subsequent  analysis 
that  start  and  end  times  of  the  scenario  segments  of  interest  are  readily  identified  on  the 
archived  data  sets.  Such  identification  may  be  time  and  date  related,  or  based  on  transmission 
of  recognizable  messages  or  orders.  These  archived  data  may  be  analyzed  later  at  any  of 
several  levels,  according  to  priorities  established  at  the  time.  These  levels  include: 

•  Frequency  of  message  type  by  sender  and  receiver  according  to 
different  phases  of  the  exercise  scenario. 

•  Message  content  and  length,  etc. 

•  Transmission  errors. 
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3.  Data  Handling  and  Analysis 

3.1.  Equipment  Required 

The  following  equipment  should  be  provided  to  each  observer  team. 

•  Radio  communication  with  the  Trial  Director  and  among  team  members. 

•  Clip  board  and  field  message  pad  for  each  observer. 

•  Data  capture  sheets. 

•  Questionnaires. 

•  Interview  and  Focus  Group  guidelines. 

•  Still  camera  and  film  for  each  Observation  Team. 

•  One  video  camera  to  be  used  as  the  Trial  Director  determines. 

•  CF-27  laptop  computer  and  portable  printer. 

•  Office  Stationary  products  (PP&S) 

3.2.  Data  Storage 

Sufficient  waterproof  data  storage  envelopes  (or  other  suitable  means)  should  be  provided  to  each 
Observation  Team  to  last  during  the  trial  (e.g.  Tyvek  type  courier  envelopes).  These  envelopes 
should  be  clearly  marked  with  the  data  type  to  be  stored  in  each  envelope  and  a  return  address  (in 
case  of  loss).  Envelopes  should  be  collected  from  each  team  at  frequent  intervals,  checked  for 
completeness  and  stored  in  a  safe  place. 

In  the  event  that  message  traffic  can  be  captured  and  archived,  then  appropriate  electronic  storage 
should  be  provided. 

3.3.  Data  Reduction 

Clerical  support  will  be  required  for  data  entry.  Prior  to  data  reduction,  a  data  entry  coordination 
meeting  should  decide  the  format,  software  and  organization  of  data  entry. 

3.4.  Data  Analysis 

The  small  number  of  trial  participants  will  preclude  the  use  of  analytical  statistics  and  data  analysis 
will  be  limited  to  descriptive  statistics,  interpreted  by  inspection. 

Analysis  of  any  archival  data  should  be  determined  at  this  point. 

3.5.  Trial  Report 

The  trial  report  is  recommended  to  contain  the  following  elements: 

•  Main  body 

with  background,  outline  method,  main  results  by  data  set,  discussion  drawing  out 
themes  which  run  across  data  sets,  conclusions  and  recommendations. 

•  Annexes  for  each  data  set  (Exit  questionnaire,  observer  comments,  etc) 
with  detailed  method,  results  and  conclusions  for  the  data  set  in  question. 
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Annex  A:  SAS  Outline  Functionality 


This  Annex  contains  and  outline  of  SAS  functionality.  'It  is  to  be  used  as  a  supplement  to  the 
Observer  Checklist  found  in  Annex  D:  Observer  Data  Package. 
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Group  A  -  Set  up: 

-  Initialize :  permits  users  to  log  on  and  set  up  the  system. 

-  Configure :  permits  users  to  configure  the  system  according  to  SOP  or  individual  choice  for  a  range  of  features. 
Group  B:  Situation  Awareness: 

-  Set  SA  map:  permits  users  to  set  up  the  electronic  map  environment  required. 

-  Maintain  SA:  permits  users  to  manage  display  of  blue  and  red  force  position  and  movement  actual  or  predicted. 

-  Objects:  permits  users  to  create,  display  and  manage  symbols  on  the  map  display. 

Group  C  -Communications: 

-  Messages:  permits  users  to  view,  check,  log,  distribute,  annotate  text  messages. 

-  Requests,  Returns,  Alerts:  permits  users  to  create  and  send  pre-formatted  messages  of  various  types. 

Group  D  -  Plan  and  Direct: 

-  Plans:  permits  users  to  create  and  send  pre-formatted  plans,  annexes,  appendices,  and  overlays. 

-  Orders:  permits  users  to  create  and  send  various  pre-formatted  orders. 


Annex  B:  Trial  Participants 


This  Annex  contains  a  table  listing  the  potential  distribution  of  SAS  for  the  exercise  by  participant, 
call  sign,  likely  other  users  etc.  This  information  is  also  shown  in  schematic  format.  This 
information  is  likely  to  change  before  and  possibly  during  the  exercise  and  should  be  updated  as 
necessary. 
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Annex  B:  Trial  participants 

For  the  purpose  of  this  plan.  Trial  Participants  are  defined  as  the  users  of  full  function  SAM  systems. 
These  SAM  systems  have  been  distributed  as  shown  in  Table  B1  below. 


Grouping 

Position 

User 

C/S 

Rank 

Vehicle 

Other  likely  users 

BG  HQ 

Rover 

CO 

9T 

LCol 

LEO 

Battle  Adj  (Capt) 

Main 

Ops  0 

OA 

KS7S9HI 

M577 

CO,  Regt  2i/c,  Duty  O(s),  10 

Alt 

Ops  WO 

OB 

wo 

M577 

Trg  0 

BCP 

Bty  Comd  Arty 

Maj 

M113 

BC  Vehicle  -  Bty  Comd,  Sigx2, 
BC  Tech 

Eng 

Eng  Trp  Comd 

E19 

Lt 

M113 

HQ  Sqn 

HQ  Sqn  2i/c 

8A 

wsmm 

LSVW 

OC  HQ  Sqn 

Cbt  tm 
“A” 

Cbt  tm  comd, 

B  Coy  Comd 
(sqn  -) 

OC 

2 

Maj 

Ml  13 

Sig 

BC 

2B 

LEO 

Loader 

FOO 

Gil 

E22HI 

Foo  Tech,  Sig 

Coy  2i/c 

I29A 

E3!9Hi 

M113 

Sig 

Cbt  tm 
“B” 

OC 

2 

Maj 

Loader 

OC 

13 

ESS9HI 

M113 

Sig 

FOO 

G12 

Capt 

M113 

Foo  Tech,  Sig 

Sqn2  i/c 

29A 

I339HI 

LSVW 

Loader 

■  ■ 

■  ■ 

BMi 

Table  Bl:  Potential  Trial  Participants. 
Anticipated  organization  of  trial  participants  by  call  sign  is  shown  below 


CO 

OA 


DCO 

OB 


BC 

G19 


1 


COTacCP 

9T 


Eng  Trp  Comd 
E19 


xsm 


BG  HQ 


OC  HQ  Sqn 
8A 


113 


SAS 


oc 

2 

B  sqn  RCD 


PLGRj 


2i/c 

29A 

B  sqn  RCD 


OC 

129 

C  coy  2  RCR 


FOO 
G11 
2  RCR 


113_ 


OC 

3 

C  coy  2  RCR 


113 _ 

SSM 

-113 _ 

CSM 

29C 

39C 

B  sqn  RCD 

j 

C  coy  2  RCR 

BC 

2B 

B  sqn  RCD 


113 - 

2i/c 

FOO 

39A 

G12 

C  coy  2  RCR 

2  RCR 

▼ 

21 

23 

B  sqn  RCD 

B  sqn  RCD 

T""  . . 

21 B 

23B 

B  sqn  RCD 

h - 

B  sqn  RCD 

Figure  Bl:  Trial  Participants  and  Potential  SAS/PLGR  Distribution 


Annex  C:  Trial  Schedule 


This  annex  repeats  the  trial  schedule  found  in  the  main  b'ody  of  the  trial  plan  and  is  to  be  used  as  a 
supplement  to  Annex  D:  Observer  Data  Package. 
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Annex  C:  Trial  Schedule 


DHH 

Activity 

im 

Exercise  commences 

i  * 

l-:2nd  May 

Independent  coy  and  sqn  training. 

Observers  coordinate  with  SAM  users.  Limited  data  capture. 

mil 

End  of  day  orders:  cbt  tm  advance  to  contact  within  BG  context  on  3-4*  May. 
Coordinate  with  SAM  users  for  observation  of  different  c/s  and  ad  hoc  interviews. 

rivv.,  -  ■ 

Advance  to  contact 

Repeat  3-5  quick  attacks  per  day  using  North-South  axis  of  Lawfield  Corridor. 

May  include  an  attack  on  bride  at  the  South  end  of  the  Lawfield  Corridor  towards  the 
end  of  day  4.  Battle  Procedure  hides  at  approximately  2300  hrs. 

Coordinate  with  SAM  users  to  complete  interim  questionnaire  on  completion. 

4*  May 

End  of  day  orders:  hasty  defence  and  delay  using  the  Jerusalem  Ridge  on  5-6*  May. 
Coordinate  with  SAM  users  for  observation  of  different  c/s  and  ad  hoc  interviews. 

Conduct  hasty  defence  and  delay  (North  end  of  Lawfield  Corridor). 

Likely  followed  by  pursuit  on  day  6. 

Coordinate  with  SAM  users  to  complete  interim  questionnaire  on  completion. 

7th  May  , 

Demonstration  Rehearsal  to  include  a  BG  attack  on  Headline  Ridge. 

This  will  be  at  least  two  hasty  attacks, 

one  with  obstacles  and  the  other  without  obstacles. 

Limited  data  capture.  Coordination  mtg  for  observer  teams.  Plan  Focus  Group 

8*May  - 

Maintenance  and  contingency  for  exercise  slippage. 

Main  Focus  Group(s)  for  user  trial  (2-3  hours). 

jjg§SB 

Visitors  day,  and  maintenance. 

Spare.  Ad  hoc  interviews  in  selected  areas,  if  required. 

Table  Cl:  Exercise  Schedule 


Annex  D:  Observer  Data  Package 


This  Annex  contains  the  materials  of  the  Observer  Data  Package.  This  includes  the: 

•  Observer  SAS  Checklist 

•  Observer  Data  Sheet 

•  Interview  Data  Sheet 

These  materials  are  to  be  used  by  Observers  during  the  SAS  trial  to  capture  Direct  Observation, 
Interview  and  Performance  Data. 
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Observer:  CIS:  Date: 

Check  scenario  type: 

Advance  to  contact  □  Quick  attack  □  Defence  □  Delay  □  Pursuit  □  Other  □ 


Notes: 

(1)  Enter  who  actual  user  is.  What  the  user  is  doing:  Planning  hasty  attack.  Making  contact  report.  Preparing  return.  Use  more  than  one  row  if  you  need. 

(2)  Enter  time. 

(3)  Enter  SAS  function  being  used. 

(4)  Enter  time  taken  to  perform  priority  SAS  tasks  (see  items  marked  *  in  SAS  checklist).  Be  as  precise  as  you  can,  Note  start  and  end  point. 

(5)  Enter  anything  that  distinguishes  (e.g.  condition,  day/night  etc.).  Use  more  than  one  row  if  you  need.  0#  =  observer  comment.  S#  =  trial  participant  comment 


Instructions:  Interview  during  lull  in  exercise.  Enter  data  indicated.  Keep  one  set  of  sheets  for  each  exercise  day. 


Observer: 


CIS: 


Date: 


Check  scenario  type: 

Advance  to  contact  □  Quick  attack  □  Defence  □  Delay  □  Pursuit  □  Other  □ _ 

Probe  questions  to  interviewee. 

1 .  What  did  you  have  to  do  during  the  past  exercise  phase? 

2.  What  did  you  use  SAS  for  during  this?  Which  SAS  features  did  you  use? 

3.  Was  SAS  was  “Useful"  for  those  tasks? 

4.  Was  SAS  was  “Easy  to  Use"  for  those  tasks? 

5.  Was  SAS  was  “Better”  than  the  existing  system  for  those  tasks?  (Faster,  less.work,  more  accurate,  etc)? 

6.  Was  SAS  “Reliable”  for  those  tasks? 

7.  Was  SAS  “Easy  to  Learn”  for  those  tasks? 

8.  Do  you  agree  that  the  training  you  received  for  using  SAS  for  those  tasks  was  “As  effective  as  possible”? 

Did  the  SAS  training  you  had  prepare  you  for  what  you  had  to  do?  How  could  the  training  have  been  improved? 

9.  Is  there  anything  that  should  be  considered  for  unit  SOPs  for  those  tasks? 

1 0.  Was  there  anything  you  chose  not  to  use  SAS  for,  even  though  the  feature  exists,  and  if  so,  why? 


1:  Exercise  Phased 


2:  :SAS  feature 

-  used? 


3-1 0:  Comments  from  interviewee  r~  - 

(How  better,  worse,  SOP  issues,  training  suggestions. 


Annex  E:  Mid  Trial  &  Exit  Questionnaire 


This  Annex  contains  Personal  Information  Questionnaire,  Mid  Trial  Questionnaire  and 
Questionnaire  Administration  Guidelines. 


The  Personal  Information  Questionnaire  should  be  completed  once  by  every  fully  functional  SAS 
user  who  completes  a  Mid  Trial  Questionnaire  and/or  a  Final  Exit  Questionnaire  or  has  used  SAS 
and  participates  in  a  mini  or  final  focus  group. 


The  Mid  Trial  and  Exit  Questionnaire  should  be  administered  according  to  the  Trial  Schedule 
(Annex  C)  or  as  indicated  by  the  Trial  Director. 
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Instructions:  To  be  completed  on  the  first  day  of  the  exercise  by  all  SAM  users  (fully  functional  SAM  systems). 


Background 


Name  /  rank: 


e.g.  1  Tp,  B  sqn,  RCD 


Posn  for  Ex: 
e.g.  Duty  O 


Years  in  CF: 


m 


e.g.  Tp  Ldr,  Bosnia 


e.g.  6  months 


Frequency  of  use  (check  one  bo 

*) 

Computer  Experience 

Never 

Seldom 

Now  &  then 

Frequently 

Always 

Desk  top  computers 

□ 

□ 

□ 

□ 

□ 

Lap  top  computers 

□ 

□ 

□ 

□ 

□ 

Windows  3.1/95/98 

□ 

□ 

□ 

□ 

□ 

Spreadsheets 

□ 

□ 

□ 

□ 

□ 

Word  processors 

□ 

□ 

□ 

□ 

O 

Drawing  or  map  tools 

□ 

□ 

□ 

□ 

□ 

Keyboards 

□ 

O 

□ 

□ 

□ 

Mouse,  trackball,  etc. 

□ 

□ 

□ 

□ 

□ 

Internet 

□ 

□ 

□ 

□ 

□ 

Electronic  mail 

□ 

□ 

□ 

□ 

□ 

Video  games 

□ 

□ 

□ 

□ 

□ 

Frequency  of  use  (check  one  bo 

<) 

SAS  Experience 

Never 

Seldom 

Now  &  then 

Frequently 

Always 

Before  this  exercise 

□ 

□ 

□ 

□ 

□ 

Name  /  rank: 

C/S  for  EX: 

Posn  for  EX: 

Date: 

e.g.  OA 

e.g.  Duty  0 

Instruction:  Check  EX  scenario(s)  covered  by  this  questionnaire. 

Advance  to  contact  □  Quick  attack  □  Defence  □  Delay  □  Pursuit  mother  □ 


Instructions:  Indicate  how  often  you  used  the  different  parts  of 
during  the  phases  of  the  exercise  today  using  the  scale  on  the  rid 


A  O 


Exercise  phase 

SAMAP 

MESSAGING/ 

COMMUNICATION 

S 

PLANS  & 
ORDERS 

1 

2 

3 

4 

5 

1 

1  2 

3 

5 

1 

2 

3 

4 

5 

Planning 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Orders 

□ 

□ 

□ 

o 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Final  Preparation 

□ 

□ 

o 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Execution 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Consolidation 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Other 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Other 

□ 

□ 

□ 

□ 

□ 

□ 

n 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

o 

Other 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

OVERALL 

□ 

o 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Instructions:  Use  the  rating  scale  on  the  right  to  do  the  following: 

-»  For  each  statement,  check  the  number  box  that  matches  your 
answer.  Only  answer  for  SAS  items  YOU  have  used  today.  Agree 
in  the  final  column  (SAS  Better?)  if  you  think  SAS  is  better  than  the 
existing  system  or  method.  Check  “NA”  if  not  used. 


The  following  SAS  functions  were: 

Useful? 

Easy  to  Use? 

SAS  Better? 

SET  UP  AND 

CONFIGURE 

©  '  @ 

BH 

Unitialize  (log  on,  reset) 

NA 

□  □ 

□  □ 

□  □ 

□  □ 

□  □ 

□  □ 

□  □ 

□  □ 

□  □ 

□  □ 

□  □  □  □  □ 

□  □  □  □  □ 

2.Configure 

NA 

SAS  training  for  these  functions  was: 

NA 

□ 

□ 

□ 

□ 

□ 

Comments  on  Set  up  and  Configuration?  Refer  to  functions  by  question  number  &  use  back  of  sheet  for  more  space. 


The  following  SAS  functions  were: 

Useful? 

Easy  to  Use? 

SAS  Better? 

SA  MAP  MANAGEMENT 

m 

m 

EH 

Is 1 

3.Graphics  management(arrows,  boundaries,  etc) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

4.Unit  symbol  management  (paste,  edit  delete, 
etc) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

5.0rientation  (centre,  scroll,  grid  lines,  heading, 
etc) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

6.Track  histories 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

7.Unit  status 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

8.Range  fans 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

9.0rbat  management 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

lO.Map  update 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

1 1  .Own  Station  Posn  Report  (OSPR) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

12.Consolidated  Position  Report  (CPR) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

SAS  training  for  these  functions  was: 

NA 

□ 

□ 

□ 

□ 

□ 

Comments  on  SA  Map  Management?  Refer  to  functions  by  question  number  &  use  back  of  sheet  for  more  space. 


The  following  SAS  functions  were: 

Useful? 

Easy  to  Use? 

SAS  better? 

Message  management 

© 

V;. 

i.;>  : .  .j 

© 

© 

© 

© 

.© 

13.View  messages  (incoming,  outgoing,  partial 
etc.) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

14.Net  management  (overhear,  affiliate,  view  net 
chart) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

15. Comm  check 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

16.  Journal 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

17.Distribution  management  (lists,  forwarding 
rules) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

18. Create  Reauests,  Returns,  Alerts 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

1 9. Send  Reauests,  Returns,  Alerts 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

20.Contact  Wait  Out 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

SAS  training  for  these  functions  was: 

NA 

□ 

□ 

□ 

□ 

□ 

Comments  on  SAS  Message  Management?  Referto  functions  by  question  numbers  use  back  of  sheet  for  more  space. 


The  following  SAS  functions  were: 

Useful? 

Easy  to  Use? 

SAS 

better? 

Plans  and  orders 

©  . 

.  .7 

©: 

© 

© 

© 

© 

21  .Create  Plans  (Plan,  Annex,  Appendix,  Overlay) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

22.Create  Orders  (Opord,  Frag  o,  mov  o,  wng  o,  coa) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

23.Receive  &  View  Plans  and  Orders 

NA 

□ 

□ 

P 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

24.Forward  Plans  &  Orders 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

25.Create  Overlays 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

26.0bject  editing  (select,  drag,  copy,  paste  etc.) 

NA 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

SAS  training  for  these  functions  was? 

NA 

□ 

□ 

□ 

□ 

□ 

Comments  on  SAS  Plans  and  Orders?  Refer  to  functions  by  question  number  &  use  back  of  sheet  for  more  space. 
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Instructions:  This  questionnaire  covers  SAS  training  and  your  experience  with  SAS  over  the  whole  exercise, 
if  you  have  not  already  done  so,  please  complete  a  Personal  Information  sheet. 


Name  /  rank: 


C/S: 


Date: 


Instructions:  Check  Exercise  Scenario(s)  completed  using  SAS.: 


Advance  to  contact  □  Quick  attack  □  Defence  □  Delay  □  Pursuit  mother  □ 


Instructions:  1:  Please  rate  SAS  use  during  this  exercise. 

1:SAS 

2:  SAS  Better? 

2:  Please  compares  SAS  with  the  existing  way  of  working.  ' 

Complete^: 

Disagree 

iwnpietay, 
. Afltes 

Conpteteiy 

Disagree 

Canpiefciy 
/gree  j 

;  "  :  ^ySuiW^^S^ngWese  Exercises: 

HI 

pm 

Iml 

H 

i. 

1  fully  tested  the  capability  of  ail  SAS  features  for: -System  configuration 

□ 

□ 

□ 

□ 

□ 

Hi 

U5S 

m 

HR 

-Situation  awareness  map 

□ 

□ 

□ 

□ 

□ 

SP| 

m 

-Message  handling 

□ 

□ 

□ 

□ 

□ 

iiii 

Hi 

HJ23 

SM 

Hi 

-Plans  and  orders 

□ 

□ 

□ 

□ 

□ 

im 

EM 

m 

n 

2. 

SAS  provided  all  the  information  1  needed  to  visualize  the  battlefield. 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

3. 

SAS  kept  me  fully  aware  of  the  location  of:  -Units  in  my  cbt  tm 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-Flank  units 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-Logistic  support  units 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-Battle  group  HQ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

4. 

SAS  allowed  me  to  be  fullv  aware  of  friendly  force:  -capabilities  /  status 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

5. 

SAS  allowed  me  to  be  fully  aware  of  enemy  force:-locations 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-capabilities  /  status 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

6. 

SAS  gave  me  complete  control  over  the  information  1  needed. 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

7. 

SAS  provided  me  with  too  much  information 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

8. 

SAS  provided  me  with  too  little  information 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

9. 

SAS  displayed  information  in  the  way  1  needed 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

10. 

SAS  readily  allowed  me  to  enter  information  /  data 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

11. 

SAS  message  handling  features  were  completely  satisfactory 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

12. 

SAS  plans  and  orders  features  were  completely  satisfactory 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

13. 

SAS  reports,  returns,  and  alert  features  were  completely  satisfactory 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

14. 

SAS  let  me  detect  battlefield  chanqes  about  friendly  units  -quickly 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-accurately 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-completely 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

15. 

SAS  let  me  understand  battlefield  chanqes  about  friendly  units:-quicklv 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-accurately 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-completely 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

16. 

SAS  let  me  communicate:  -quickly 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-accurately 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-completely 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

17. 

SAS  helped  me  to  make  good  decisions 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

18. 

SAS  helped  me  to  make  quick  decisions. 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Comments 


;  ~  write  in  your  answers  •" 

BENEFITS 

The  three  biggest 
benefits  of  SAS  see 
are: 

1. 

2. 

3. 

t  * 

CONCERNS 

My  three  biggest 
concerns  with  SAS 
are 

1. 

2. 

3. 

Please  rate  the  SAS  training  course  you  received. 


If  you  did  not  attend  the  SAS  training  course, 
check  this  box  □  and  go  to  the  next  section. 

Co ospiksiy .  "Vi 
Disagree  /, . 

Completely 

Agree 

©  •" 

::p:; 

Any  Comments? 

1. 

1  felt  completely  ready  to  use  SAS  after  the  training. 

□ 

□ 

□ 

□ 

□ 

2. 

Training  aids  used  during  SAS  course  were  easy  to  understand. 

□ 

□ 

□ 

□ 

□ 

3. 

The  SAS  training  manual  was  easy  to  understand. 

□ 

□ 

□ 

□ 

□ 

4. 

There  was  enough  hands  on  practice. 

□ 

□ 

□ 

□ 

□ 

5. 

1  could  readily  relate  SAS  features  to  my  army  tasks. 

□ 

□ 

□ 

□ 

□ 

6. 

SAS  training  covered  everything  1  needed. 

□ 

□ 

□ 

□ 

□ 

7. 

SAS  training  was  long  enough. 

□ 

□ 

□ 

□ 

□ 

8. 

Overall,  1  am  satisfied  with  the  training  1  received. 

□ 

□ 

□ 

□ 

□ 

ny  overall  comments  on  SAS  training? 


Please  rate  SAS  for  HUMAN  COMPUTER  INTERFACE  issues. 


Completely 

Disagree 


© 


1. 

1  could  read  text  easily. 

□ 

□ 

□ 

□ 

P 

2. 

1  could  adjust  screen  brightness  to  suit  my  needs. 

□ 

□ 

□ 

□ 

P 

3. 

SAS  Icons  (etc)  were  easy  to  understand. 

□ 

□ 

□ 

□ 

P 

4. 

SAS  was  located  conveniently  in  my  CP  /  vehicle  t 

□ 

.□ 

□ 

□ 

P 

5. 

On  screen  colours  were  meaningful. 

□ 

□ 

□ 

□ 

P 

6. 

Menu  terminology  was  easy  to  understand. 

□ 

□ 

□ 

□ 

P 

7. 

1  was  able  to  select  from  menus  easily. 

□ 

□ 

□ 

□ 

P 

8. 

Information  was  arranged  in  a  logical  order. 

□ 

□ 

□ 

□ 

P 

9. 

It  was  easy  to  move  around  in  the  SAS  software. 

□ 

□ 

□ 

□ 

P 

10. 

1  could  easily  see  the  information  1  needed  in  each  window. 

□ 

□ 

□ 

□ 

P 

11. 

Text  was  easy  to  enter. 

□ 

□ 

□ 

□ 

P 

12. 

Numbers  were  easy  to  enter. 

□ 

□ 

□ 

□ 

P 

13. 

It  was  easy  to  edit  information. 

□ 

□ 

□ 

□ 

P 

14. 

The  information  1  needed  was  easy  to  find. 

□ 

□ 

□ 

□ 

P 

15. 

It  was  easy  to  move  between  screen  fields. 

□ 

P 

□ 

□ 

P 

16. 

The  windows  1  wanted  were  easy  to  call  up. 

□ 

□ 

□ 

□ 

P 

17. 

It  was  easy  to  know  which  window  1  was  working  on. 

□ 

□ 

□ 

□ 

P 

18. 

It  was  easy  to  arrange  windows  on  the  screen. 

□ 

□ 

□ 

□ 

P 

19. 

It  was  easy  to  operate  SAS  using  only  the  keyboard. 

□ 

□ 

□ 

P 

P 

20. 

1  could  put  the  cursor  where  1  wanted  it. 

□ 

□ 

□ 

P 

P 

21. 

1  always  knew  where  the  cursor  was  on  the  screen. 

□ 

□ 

□ 

P 

P 

22. 

It  was  easy  to  get  to  the  “Help”  function. 

□ 

□ 

□ 

P 

P 

23. 

SAS  controls  were  always  within  easy  reach 

□ 

□ 

□ 

P 

P 

24. 

“Help"  information  was  useful. 

□ 

□ 

□ 

P 

P 

25. 

SAS  responded  quickly  enough  to  my  commands. 

□ 

□ 

□ 

P 

P 

26. 

SAS  use  was  always  compatible  with  other  tasks 

□ 

□ 

□ 

P 

P 

Any  overall  comments  on  SAS  HCI  or  workstation  configuration? 


I 


ANY  OTHER  COMMENTS? 


fuestimnairesAdmifiimratiommelines 


ip.pp.ss 


Annex  E  (continued):  Questionnaire  Administration  Guidelines 


•  The  Observer  Team  should  themselves  complete  all  questionnaires  as  a  group  and  agree  on  a  common  interpretation 
of  the  questions.  This  must  be  done  so  that  the  Team  is  prepared  for  administering  the  questionnaire  (briefing  and 
answering  questions  etc.) 

•  Each  Observer  Pair  should  brief  participants  on  the  intent  of  the  questionnaire  with  particular  attention  to: 

•  The  interchangeability  of  SAS  and  SAM 

•  The  focus  of  each  major  heading  e.g.  what  “Message  Management”  encompasses 

•  The  differences  between  Useful,  Easy  to  Use,  and  SAS  Better 

•  Encouraging  honest  and  constructive  feedback 

•  Each  Observer  Pair  should  ensure  participants  understand  the  importance  of  the  context  of  their  SAS  feedback  i.e. 
which  phase  of  the  scenario  (e.g.  advance  to  contact,  delay)  and  which  sub-phases  (e.g.  preparing  for  the  attack  or 
actually  conducting  the  attack).  It  is  important  that  participants,  understand  and  are  able  to  differentiate  these  sub¬ 
phases  when  giving  verbal  or  written  feedback. 

•  Participants  should  be  encouraged  to  ask  any  questions.  Special  attention  should  be  paid  to  assisting  French 
speaking  participants  with  potential  technical  wording  difficulties. 

•  Participants  should  be  encouraged  to  write  comments  in  boxes  provided  or  on  the  back  of  the  sheets. 


Annex  F:  Focus  Group  Guidelines 

This  Annex  contains  Exit  Trial  Questionriaires  and  Focus  Group  Guidelines 


Humansyste/ws  Incorporated 


Annex  F 


Annex  F  (continued):  Guidelines  for  Focus  Groups  and  Interviews 

Both  interviews  and  focus  groups  are  methods  for  eliciting  information  from  people  on  a  particular  topic  within  a  limited 
time.  This  requires  a  framework  to  make  best  use  of  the  time  available  and  to  provide  the  interviewee  or  the  focus  group 
members  with  a  clear  idea  of  what  is  required.  There  also  need  to  be  ground  rules  to  ensure  that  they  feel  free  to  speak, 
but  not  to  ramble  or  indulge  themselves  too  much  with  long  war  stories  (fascinating  though  these  may  be). 

Both  approaches  require  preparation. 

Focus  Groups 

For  this  trial,  two  sorts  of  focus  groups  are  envisaged. 

•  A  main  focus  group  at  the  end  of  the  exercise  with  all  trial  participants  present. 

•  A  mini-focus  group  run  as  and  when  possible  during  the  exercise  -  probably  only  once,  at  approximately  the  mid 
point  of  the  trial.  The  mini-focus  group  would  only  comprise  the  members  of  one  or  other  combat  team,  or  battle 
group  HQ,  not  the  whole  group  of  trial  participants. 

For  the  main  exit  focus  group,  the  Trial  Director  should  hold  a  formal  coordination  meeting  of  the  observer  teams  to 
determine  responsibilities  and  finalize  format  and  facilities.  Mini  focus  groups  will  be  less  rigorous  and  much  shorter  but 
should  follow  the  same  general  pattern,  as  far  as  conditions  permit. 

Facilities 

The  main  exit  focus  group  will  require  at  least  two  hours  and  should  be  conducted  under  shelter  and  away  from 
distractions.  A  class  room  environment  would  be  ideal,  though  not  essential.  One  observer  should  facilitate  discussion 
while  the  other  acts  as  a  recorder  taking  down  comments.  A  black  or  white  board,  and  flip  chart  sheets  tacked  to  a  wall 
are  a  good  way  of  recording  comments,  posting  ground  rules,  etc. 

Format 

Focus  group  format  should  combine  SAM  functionality  with  the  exercise  sequence  to  review  users  experiences  while 
systematically  using  probe  questions.  Supplementary  questions  may  arise  as  a  result  of  observer  experience  during  the 
exercise. 

Agenda 

This  can  vary  but  should  cover  the  following  points. 

•  Welcome  and  introductions 

•  Goal 

•  Structure 

-  Briefly  review  the  structure  and  function  of  SAM. 

(Post  an  outline  as  a  cue  for  participants  during  discussions). 

-  Briefly  review  the  exercise  scenarios  covered. 

-  Highlight  the  probe  questions  that  will  be  asked. 

•  Conduct  the  focus  group 

-  For  each  type  of  exercise  scenario  conducted  step  participants  through  the  various  stages  of  the  scenario,  and  for 
each  scenario  stage  prompt  for  comments  on  use  of  SAS  functions.  Allow  some  discussion,  encourage  everyone 
to  speak,  but  maintain  the  pace. 

•  Consensus 

-  Ask  each  member  to  write  down  the  best  three  and  worst  three  things  about  SAS,  from  their  point  of  view. 

Then  go  around  the  group  and  write  on  a  flip  chart  all  of  these  items.  There  will  probably  be  some  consistency 
among  the  answers.  Once  this  is  done,  take  a  vote  to  arrive  at  a  consensus  as  to  the  three  best  and  three  worst. 
This  approach  is  very  subjective,  but  helps  focus  discussion  and  will  often  unearth  significant  insights. 

Ground  rules 

Any  focus  group  should  start  with  an  outline  of  the  ground  rules.  These  rules  are  like  brainstorming  rules.  The  intention 
is  to  maximize  participation  and  minimize  any  chilling  effect  resulting  from  judgmental  or  dismissive  statements  from 
authority  figures.  For  instance,  very  senior  officers  or  members  of  the  SAM  design  team  should  not  be  permitted  to 
participate  during  the  focus  group  though  observation  and  discussion  with  participants  afterwards  can  be  permitted.  Focus 
group  facilitators  should  balance  the  tendency  of  dominant  personalities  of  any  rank  to  take  over  the  commentary  by 
encouraging  less  dominant  personalities  to  speak  first.  The  idea  is  to  get  the  ideas  out  and  recorded.  Judgement  can  come 
later.  Have  zero  tolerance  for  statements  that  put  down  someone’s  comment.  Instead  try  to  understand  what  lies  behind 
the  comment.  On  the  other  hand,  be  disciplined  about  time.  The  tempo  must  be  maintained  or  all  the  ground  will  not  be 
covered.  A  common  error  is  to  have  too  much  discussion  early  on  during  the  focus  group  while  everyone  is  fresh,  and 
having  to  rush  the  later  topics.  Ban  time  consuming  war  stories. 
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Interviews 

During  this  trial,  interviews  will  be  need  to  be  conducted  ad  hoc  when  time  can  be  won  from  the  exercise 
schedule.  This  time  may  be  quite  short,  and  broken.  Opportunities  are  likely  to  occur  at  short  notice.  Be  careful 
not  to  intrude  on  the  military  responsibilities.  A  positive  personal  rapport  between  interviewer  and  interviewee  is 
very  important.  The  interviewer  must  win  over  the  interviewee(s)  by  showing  genuine  interest  in  their  opinions 
and  consideration  for  their  priorities. 

There  is  little  point  in  conducting  any  interviews  until  trial  participants  have  had  experience  with  SAS.  On  the 
other  hand,  an  interview  conducted  soon  after  a  particular  exercise  scenario  such  as  an  advance  to  contact  or 
quick  attack,  will  mean  that  the  experience  is  fresh  in  the  interviewees  mind. 

The  observers  data  sheet  provides  the  framework  for  ad  hoc  interviews.  Like  the  focus  group,  the  goal  is  to 
provide  a  framework  that  will  let  the  interviewee  share  their  insight  in  a  way  that  systematically  covers  the 
ground  in  a  limited  time.  Lay  out  the  ground  rules,  then  listen,  probe  and  record.  Avoid  giving  your  own 
opinion  or  judging  their  comments. 

Review  the  interview  data  sheet.  Systematically  step  the  interviewee  through  their  experience  with  SAS  to  date  by  exercise 
phase.  Use  the  SAS  function  check  list  to  prompt  yourself  and  them  about  different  features  to  confirm  that  they  did  or  did 
not  use  them.  Build  further  supplementary  prompt  questions  as  you  go  along,  but  be  careful  not  to  lead  the  interviewee  or 
you  are  likely  to  get  the  answer  he  or  she  thinks  you  want. 

DO: 

•  Set  up  a  mutually  agreed  understanding  of  when  and  where  you  can  interview  without  getting  in  the 
way  or  interfering  with  their  tasks. 

•  Brief  the  user  on  what  it  is  that  you  are  trying  to  achieve. 

•  Ensure  the  user  knows  that  you  are  there  to  ask  questions  and  record  answers  only  (good  or  bad)  and 
not  to  advocate  or  defend  SAS. 

•  Ensure  that  the  user  know  you  are  not  evaluating  any  part  of  their  own  performance  -  only  SAS. 

•  Ensure  the  user  knows  you  are  not  a  SAS  assistant  and  are  not  there  to  help  with  SAS. 

•  Use  the  Interview  data  sheet. 

•  Ask  neutral  but  specific  contextual  questions.  (Do  not  ask,  “ How  did  you  like  the  map  ”.  Instead  ask, 
“Did  you  use  the  map  during  the  advance?”,  “Was  the  scale  appropriate  when  advancing?  Did  you 
have  to  change  the  scale?  ”,  “What  were  the  lighting  conditions  and  did  they  affect  the  use  of  the 
map?”,  “Was  there  a  problem  (or  benefit)  with  the  map  during  the  advance?  How  might  an  identified 
problem  be  fixed?”  Avoid  double  barreled  questions  -  these  can  be  confusing.) 

•  Ask  users  about  their  problems  with  SAS  and  about  the  benefits,  potential  or  actual. 

DON’T: 

•  Do  not  ask  leading  questions  (For  example,  do  NOT  say  anything  like:  “The  CO  really  seems  to  like 
function  x,  and  what  do  you  think  about  that?  Really  dumb  eh!”  Instead,  ask  something  neutral  and 
then  probe  for  more  “Why  ”  insight?  Then  record  the  answers  without  comment.  You  are  seeking  data, 
not  conclusions.) 

•  Do  not  argue  with  users,  instead  probe  until  you  clearly  understand  the  root  cause. 

•  Do  not  correct  users  or  point  out  what  they  are  doing  wrong.  Probe  for  a  full  explanation  of  the 
problem  and  what  they  think  should  be  done  to  fix  it. 

•  Do  not  defend  the  design  or  explain  the  problem  away.  (For  example,  if  a  user  says,  “It’s  too  slow”, 
do  not  say,  “Yes,  we  know  about  that  and  it  is  being  fixed”.  Instead,  ask,  “Why  ”,  and,  “How  fast  is 
fast  enough  "  etc.  Never  dismiss  any  feedback.) 

All  of  the  “DON’Ts”  above  are  crucial  to  maintaining  the  flow  of  feedback.  Violating  any  of  them  will  serve 
to  make  the  users  “clam  up”  or  worse  yet  give  you  data  bias  -  the  user  may  say  the  system  is  really  good  in 
hope  that  you  will  go  away  or  really  bad  in  an  attempt  to  “get  even”. 
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SAM  contains  a  number  of  automatic  message  logging  and  screen  saving  and  archiving  functions. 
These  may  be  used  to  save  data  for  subsequent  baseline  analyses.  Examples  include: 

•  plotting  of  message  frequency  by  type,  exercise  phase  and  unit. 

•  Comparing  task  performance  times  (such  as  provision  of  requested  reports  and  returns)  using 
SAM  with  performance  times  taken  on  other  occasions  for  similar  tasks  when  not  using  SAM. 


